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Introduction 



The purpose of the present study was twofold. The first objective was to compare three common 
measures of agreement in describing child behavior problems. Pearson correlations, Cohen's kappa, and 
percent agreement calculated from die data were compared in order to examine characteristics of each 
measure in describing interinformant agreement for ratings of child behavior. Researchers have 
calculated agreement differently thus complicating comparison across studies. Some of the putative 
measures of agreement suffer from flaws that are not recognized or acknowledged (e.g., Bartko & 
Carpenter, 1976), and researchers often make assumptions about the appropriateness of measures with 
little rationale (e.g., Berry & Mielke, 1988). The blind use of an analytical tool without acknowledgment 
of its shortcomings and appropriateness for measuring the construct of agreement in a specific situation 
may produce results that are misleading as to the actual degree of agreement between informants. 

Typically, agreement has been expressed using product-moment correlations of standardized summary 
scores across pairs of informants (Achenbach, McConaughy, & Howell, 1987; Eisenstadt, McElreath, 
Eyeberg, & McNeil, 1 994) or correlations across items for pairs of informants (Achenbach, 1991; 
Fischer, Barkley, Fletcher, & Smallish, 1993). Although correlations reflect relative association between 
informants, they are not appropriate indices of interinformant agreement (Bird, Gould, & Staghezza, 
1992). Correlations reflect only the association between groups of items and not absolute agreement 
between raters. It is possible to obtain a high degree of association (correlation) yet little agreement 
within the pair (Jensen, Traylor, Xenakis, & Davis, 1988). The pattern of responses within a pair of 
informants may be almost identical yet be anchored at different points on a scale. 

An alternative index of agreement is kappa, first proposed by Cohen (1960), which assesses the 
proportion of agreement after removing die effects attributable to chance. It too has been criticized. Not 
only is the statistic tedious to compute, its values vary with factors other than agreement, specifically 
sensitivity, specificity, and the prevalence or base rate of the target illness or behavior. Several 
researchers have acknowledged some of these problems (Brennan & Hays, 1992; Feingold, 1992; 
Hutchinson, 1993; Maclure & Willett, 1987) including Spitznagel and Helzer (1985) who noted that the 
problems caused difficulty in comparing kappa values across studies. Even so, researchers have not 
adequately recognized and addressed the impact of varying base rates on kappa within studies. 

Finally, an expression of interinformant agreement that avoids the problems that plague kappa and 
correlation is percent agreement. Percent agreement is relatively easy to calculate, and researchers and 
practitioners alike comprehend its meaning. Recently, more researchers investigating interinformant 
agreement of child behavior ratings have utilized percent agreement as the descriptor of choice (Kaslow, 
Warner, John, & Brown, 1992; Kolko & Kazdin, 1993; Renouf & Kovacs, 1994). Although the statistic 
does not account for chance agreement, the argument is made here that when considering agreement 
between experts' ratings, chance agreement is inconsequential. The rationale needs to presented for 
making deductions rather than vice versa. 
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The second objective of this study was to examine several child, parent, and dyadic-level variables for 
significant correlations with mother/father agreement about their child's behavior. Particular emphasis 
was placed on looking at dyadic-level variables that have received scant attention in the literature 
relative to individual-level variables such as gender and child age. 

Return to Top 



Method 



Participants were children and their families who were referred to a pediatric clinic for assessment of 
common behavioral concerns and for whom Child Behavior Checklist (CBCL; Achenbach, 1991) data 
were available from both parents. Sixty children, 46 males and 14 females, ranging in age from 4 to 12 
years (M = 6.7, SD = 2.05), comprised the sample. Ninety percent of the sample was Caucasian. All 
lived with two parents, 80% with both biological parents while the remaining children's parents were 
divorced and remarried. Finally, participants represented a wide range of socioeconomic levels. In 
addition to the CBCL, parents completed a variety of measures including the Duncan Socioeconomic 
Index (Duncan, 1961), the Mental Health Inventory (MHI; Veit & Ware, 1983), the Dyadic Adjustment 
Scale (DAS; Spanier, 1976), and the Life Experiences Survey (SLES; Sarason, Johnson, & Siegal, 
1978). 
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Results 



Measures of Agreement 

Figure 1 demonstrates that, given the same number of disagreements, kappa values, and to a lesser 
extent correlations, decreased with higher mother/father pair base rates, or more agreement that the 
behaviors did not describe their child. For children who exhibit few problem behaviors, parents are more 
likely to have relatively more response combinations in one cell (0,0 for the CBCL). Because of kappa's 
calculation and definition of chance, when parents agreed that a substantial number of items did not 
characterize the behavior of their child, a larger deduction in the pair's kappa value was made to account 
for chance, thereby reducing the observed kappa value. Thus, over pairs, there is greater variance in the 
measure resulting from factors not related to agreement. When used as a dependent measure, the 
increase within measure variance of kappa and correlations due to marginal symmetry and higher base 
rates make them far less sensitive to the independent variables. In addition to Figure 1, this fact also is 
demonstrated in the tables by fewer significant findings when using kappa or correlations as a measure 
of agreement. 

We recommend using percent agreement as a measure of agreement between respondents when they are 
providing judgments about which they can be considered an expert, and particularly when one response 
category might be more frequently used. Percent agreement is easier to calculate and understand 
conceptually than are other measures of agreement. As mentioned above, it also can be more powerful if 
other measures suffer from large within-measure variance. 

One criticism of percent agreement as a measure of agreement is that it does not account for agreement 
due to chance. However, we assert that chance has very little impact in observed agreement when 
respondents are asked to make judgments about which they can be considered experts. Parents make 
expert judgments about their children's behavior over time. Thus, the similarity between informants' 
responses results almost solely from similar perceptions rather than from chance. 

Factors Related to Mother/Father Agreement 

Average agreement between mothers and fathers was high, 73.5% (SD = .10, range 40.7% - 94.8%). 
Percent agreement correlated with several child and parent characteristics including mothers' and fathers' 
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levels of psychological distress, general well-being, and reported recent positive life events. Agreement 
also was negatively related to fathers' reported negative life events and positively related to mothers' 
reported SES. 

Dyadic psychological distress and general well-being were significantly correlated with agreement. 
Couples with more distress showed less agree-ment, and couples with higher well-being scores showed 
more agreement. Mother/father agreement on their children's behavior was negatively correlated with 
discrepancy scores for marital satisfaction, affection, consensus, and negative life events. Our findings 
emphasize the need to look at dyadic- and family-level variables in addition to individual characteristics. 
Please refer to Table 1 and Table 2 for significant correlations between agreement and individual and 
dyadic-level variables. (Although percent agreement is supported here as the index of choice, results 
using kappa and correlations are retained for illustrative purposes as well as to compare to other studies.) 
Return to Top 



Implications 



On average, we found a high degree of agree-ment between parents when reporting on their children's 
behavior. This finding is important in relation to screening. Because of the similarity in mothers' and 
fathers' reports, screening can be undertaken using the report of only one parent. However, once a child 
is identified or when a more in-depth assessment is made, information should be collected from both 
parents to provide a more complete picture of the child and family context. Additionally, our findings 
emphasize the need to look at dyadic- and family-level variables in addition to individual characteristics. 



In the future, more research should investigate the outcomes of agreement. For example, do children 
whose parents exhibit higher agreement demonstrate better outcomes? Is higher mother/father agreement 
related to treatment compliance or remaining in treatment? An interesting finding of this study was that 
the number of disagreements about the presence/absence of a behavior was more than twice the number 
of disagreements about the frequency of a behavior. Future research should investigate the hypothesis 
that better outcomes would be exhibited by children whose parents disagreed more about the frequency 
of behaviors than about the presence/absence of behaviors. 

Return to Top 
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Introduction & Method 

Intensive Case Management (ICM) for children is New York State's most intensive, home-based service 
option within its community based system of care. Intensive case managers are available to children and 
families at all times and serve a small caseload of ten children and families. They perform a 
comprehensive assessment of family and child needs and link families to needed services. Work with 
children and families is done in settings outside of the office and providers are expected to actively 
advocate for enhanced service delivery for children and families and system change. In addition, flexible 
service dollars are available for the purchase of services otherwise unavailable. 

In evaluating outcomes for children enrolled in ICM, researchers at the Bureau of Evaluation and 
Services Research of the New York State Office of Mental Health conducted a longitudinal study of 
a sample of 199 children enrolled in the program. On average, children in the sample were 1 1 years of 
age, white, non-Hispanic, mostly male, primarily lived in single parent households, were in the custody 
of a biological parent, and went to school in a special education setting. The most common diagnosis 
among these children was disruptive behavior disorder. They were out of home (either hospitalized or in 
out-of-home placement) an average of two times prior to enrollment, were functionally impaired in an 
average of 2.5 out of 5 areas, and displayed an average of 5.5 problem behaviors or symptoms (out of 
25). 

The study included a number of measures to describe the status of children and families enrolled in the 
program. To measure functioning status of both children and families, the Child and Adolescent 
Functional Assessment Scales (CAFAS; Hodges, 1990) was administered at enrollment and at three 
years or discharge. In addition to CAFAS scales, child status was also measured using a yes/no checklist 
of five areas of functional impairment, a yes/no checklist of presence of 25 behaviors and symptoms, 
and from the parent perspective, the Child Behavior Checklist (Achenbach, 1991). Additional measures 
of family status were an assessment of family problem conditions, evidence of abuse in the family's 
history, and six selected scales from the Child Well-Being Scales (Magura & Moses, 1986). The 
availability of this set of child and family functioning measures allowed for an examination of the 
validity of the CAFAS measure for this population of children with serious emotional disturbances and 
their families. 

A factor analysis of the CAFAS scales produced two discrete factors. The CAFAS subscales fell cleanly 
into a Child Functioning factor (Child CAFAS) made up of the Role Performance, Behavior Towards 
Self and Others, Moods and Emotions, and Thinking subscales and a Caregiver Resources Factor 
(Caregiver CAFAS) made up of the two caregiver resources subscales. The Substance Abuse subscale 
did not load on either factor and was omitted from this investigation. These factors along with the Total 
CAFAS Score (omitting substance abuse) comprised the key measures that are examined by the analyses 
that follow. 
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Results 



Family History of Abuse 

Figure 1 displays the ability of CAFAS scales to discriminate between families that were reported as 
having evidence of various types of abuse and those that were not. For all types of abuse or neglect, the 
Caregiver CAFAS and Total CAFAS mean scores were significantly different for groups of families 
where such abuse and neglect was present, versus those that had no such evidence. In the case of sexual 
abuse, the Child CAFAS score was able to discriminate, as well. 

Family Functioning 

On measures of family problems, caregiver capacity for childcare, primary caregiver/spouse relationship, 
continuity of caregiving, caregiver's ability to recognize the child's problem condition and caregiver's 
cooperation with service provision, the Caregiver CAFAS scale significantly discriminated between 
groups of families that were clustered by high and low levels on each measure. The Total CAFAS 
significantly discriminated between groups of families that were identified with high and low levels of 
family problems, caregiver capacity for childcare, continuity of caregiving, and caregiver cooperation. 

Areas of Functional Impairment 

A yes/no checklist assessing functional impairment in five areas was administered to intensive case 
managers when children were enrolled in the program. The Child CAFAS and Total CAFAS scores 
significantly discriminated (at various levels of significance) between children identified as being 
impaired in the areas of self-care, cognitive functioning, and self-direction (see Figure 2). The two areas 
where no significant discrimination was observed have heavily skewed distributions. Almost all children 
were identified as impaired in the area of social relationship functioning and very few were impaired in 
the area of motor functioning. 

In general, CAFAS measures did not dis-criminate for the presence or absence of symptoms/behaviors 
for this population, adding to evidence that measures of functioning and symptomatology may be 
independent. Only in the area of psychotic symptomatology was the Child CAFAS scale able to 
discriminate among children who displayed these symptoms from those who did not. 

The Child CAFAS scale was able to discriminate between children who scored in the clinical range on 
the parent completed CBCL Total Problem, Internalizing, Externalizing scales. The Total CAFAS only 
discriminated on the Total Problem Score (see Figure 3). 

CAFAS and ICM Outcomes 

A logistic regression analytic model was used to examine the ability of CAFAS scales to discriminate 
between children who were hospitalized after enrollment in the program and those who remained in the 
community. This model successfully classified hospitalization outcomes for 87.3% of children (see 
Table IV Male children with high average total CAFAS scores (20 to 30), who were referred from 
mental health programs, lived in non-family settings at enrollment, and had recent contact with the 
mental health service system, had a 72% probability of being hospitalized after enrollment in ICM. This 
model identified the presence of these conditions as corresponding with the greatest likelihood of 
hospitalization. 

Shifting each measure in the model to an opposite or less severe condition, while leaving the others 
unchanged, decreased the probability of hospitalization and tested the strength of each measure's 
contribution to the prediction of a hospitalization outcome. Change in the CAFAS measure made the 
greatest impact on the prediction of hospitalization. A child with a low average CAFAS score (0 to 10) 
had a 32% probability of hospitalization when all the other measures were unchanged, a decrease in 
probability of 41% from a high CAFAS score. Shifting from a High CAFAS to a Medium CAFAS (10 
to 20) reduced the probability only by 8%. 
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Conclusion 



From the perspective of the evaluation of ICM, the correlation between the CAFAS measures and CBCL 
scores suggest a shared sense of children's problem severity among parents and providers. More 
generally, the data presented here contribute to the body of evidence that the CAFAS measure is a 
psychometrically valid measure of child and family functioning. Its attractiveness as a measure is further 
strengthened by the ease with which the scales can be successfully administered to providers and the 
resultant high response rate. 

Return to Top 



References 

Achenbach, T.M. (1991). Manual for the child behavior checklist and 1991 profile. Burlington, VT: 
University of Vermont Department of Psychiatry. 

Hodges, K., (1990). Child and adolescent functional assessment scales. Nashville TN: Vanderbilt Child 
Mental Health Services Evaluation Project. 

Magura, S. & Moses, B. S. (1986). Outcome measures for child welfare services. Washington, D.C. , 
Child Welfare League of America 
Return to Top 



Steven Huz, MPA 
Research Scientist II 
518/473-9559 Fax: 518/474-7361 
E-mail: SRHUZ@NKI.IRIS.RFMH.ORG 

Mary E. Evans, Ph.D. 

Interim Director 

518/474-7209 Fax: 518/474-7361 
E-mail: SREVANS@NKI.IRIS.RFMH.ORG 

Thomas L. McNulty, M.A. 

Research Scientist I 

518/473-6002 Fax: 518/474-7361 

E-mail: TLM238@UAL562.ALBANY.EDU 

Bureau of Evaluation and 
Services Research 
New York State Office of 
Mental Health 
44 Holland Avenue 
Albany, NY 12229 

Return to Top 




1 6 



12/11/98 1:40 PM 



8th.349f2 http://lumpy.lmhi.ust'.edu/CFSroot/rtc/proceedmg8th/8th.349t2.html 



o 



OJ 




o 




1 Of l 12/11/98 1:04 PM 



ON 

Tf 

m 

4 

00 



00 

W) 

.9 

■a 

0> 

0> 

o 

s 

cl 

o 

£ 

o 

s 

CO 

u< 

O 

T3 

<D 



-c 

£ 

& 



f 





<D 






c 

0 

1 

43 

Cl 

co 

o 

X 

S q 5 
lo 

E^ 

£§ 

© S ffi£2 

■9 ^ |=hJI 
^ w 

CD O) 

CO o 

8^ 

CD 

. 2 : 

CO 

s 



•K 



o 

Q_ 



iH 

|U| 

iff 

l*fi 


! 

It 


8-cj* 


I 

£ 

a! 

S* 


f 

o 


1? 

go 

St 5| 

ill 


I 

8 

1 

St 

Pi 

o 


1 

a 


1 yO 

! | 
| ° 


R 

St 

O 


* 

o 


Q 

K> 

o 


a 

* 

o 


& 

8 

o 


■si 

1 

*! 


1 s? 
! a 

i ^3 

i oo- 


ll 


"to 

1 

* 

B# 

w| 

oo- 


■a 

fa. 

K3 

% 

o 


a| 

|»U 

r£|| 

Sill 


Si 


i 

H 

1 


<0 

< 

8 

! 1- 
Ill 


1 

»9Jk 

i! 

IS 


Sjk 

I® 

s| 

3a 


K 

1 

<3 


M 

li 

il 

II 

aa 


ft 

|co 

*6p£ 

16 



J 

* 

b 

M 

6 

? 

I 

8 

£ 



C\? 




O 

CVJ 




1 of 1 12/11/98 1:04 PM 



8th.355 



http://lumpy.fmhi.usf.edu/CFSroot/rtc/proceeding8th/8th.355.html 



I Home 1 1 search 1 1 index! 

8th Annual Research & Training Center Conference Proceedings, Dept of Child and Family Studies, 
Florida Mental Health Institute, University of South Florida, 1996 

The CAFAS as an Outcome Measure in the Alternatives to 
Residential Treatment Study 



Author 

Introduction Method Results References 
Back to Table of Contents 



Introduction 

Psychiatric nomenclature advances a multiaxial approach to psychiatric assessment emphasizing 
measurement of five dimensions: psychiatric symptoms, personality disorders, physical health problems, 
type and severity of life stressors, and global assessment of functioning. This last dimension has evaded 
sophisticated measurement; however, development of tools to measure global functioning is increasingly 
important as funding agencies include functional impairment in definitions of youths' serious emotional 
disturbance, and third party payers rely on severity of impairment in order to qualify persons for 
intensive services (Hodges & Gust, in press). Over the past several years, Kay Hodges has been 
developing and refining the Child and Adolescent Functional Assessment Scale (CAFAS; Hodges, 1990; 
Hodges, Bickman, & Kurtz, 1991; Hodges & Gust, in press). The 1990 version of the CAFAS was 
designed to measure impairments in youths' performance of expected roles within the family, school, 
and community. The purpose of the present study was to evaluate internal consistency and validity of the 
1990 CAFAS as well as its sensitivity to change over time. 
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Method 

Procedure 

The Alternatives to Residential Treatment Study (ARTS) was designed to describe five exemplary 
community-based alternatives to residential treatment programs 1 (detail in Duchnowski, Johnson, Hall, 
Kutash, & Friedman, 1993). All youth entering selected programs were eligible for study recruitment if 
they were between 6 and 1 8 years of age, it was their first entry into the program, and they and their 
caregivers consented within one month of entry. Demographic information, history of services received, 
multi-dimensional psychosocial functioning, and emotional/behavioral problems were assessed. Entry 
into the study included baseline, 6 month, and 1 year follow-up assessments. One hundred sixty- three 
youth participated in the first assessment. Youth who had complete data for baseline and 12-month 
follow up were included in the present study (130 youth; sample attrition from initial assessment to 1 
year follow-up 20.2%). There were no differences on age, race, sex, or symptom severity between those 
with complete longitudinal data and those without complete data. 

Participants 

Youth averaged 14.1 years (SD = 3.1 years). Sixty-six percent were male, 64.6% Caucasian, 12.3% 
African American, 12.3% Native American/Alaskan, 10.0% Latino, and one youth was Asian. At the 
point of entry into the programs, these youth presented emotional, behavioral, and social problems that 
were severe, long term, and complex (Cascardi, Kutash, & Duchnowski, 1994). Youth had received 
prior residential mental health services an average of 4 times, 61% of the sample had prior involvement 
with law enforcement and juvenile justice systems, and nearly 80% percent had received Special 
Education services prior to program entry. Sixty-three percent entered the programs from an 
out-of-home residential setting. 
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Measures 

Baseline and 1-year follow-up data from the CAFAS, Child Behavior Checklist (CBCL; Achenbach, 
1991), and Rosenberg's (1989) brief 10-item Self-Esteem Scale, and type of services used over 1 year 
were analyzed. The CBCL is a widely-used measure utilizing parent report of presence and severity of a 
variety of emotional and behavioral problems of childhood and adolescence. It yields total, internalizing, 
and externalizing behavior problem scores. The Self-Esteem Scale is a youth report of general self 
worth. The 1990 CAFAS is a brief, multi-dimensional measure of impairment in functioning, with 5 
subscales each utilizing four rating categories. 
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Results & Discussion 

Reliability analysis revealed the first 5 subscales of the CAFAS were internally consistent, with 
a Chronbach's alpha of .7122. Table 1 presents information on validity of the CAFAS. The bivariate 
association of CAFAS total and subscales scores, CBCL, and Self-Esteem were evaluated with Pearson 
product-moment correlations. 

The CAFAS total score and subscales were significantly associated with CBCL scores. Deficits in 
youths' ability to conform their behavior to appropriate social roles was more significantly associated 
with externalizing than internalizing behavior problems (z = 2.51, p<.05). The magnitude of the 
correlations between the CAFAS and CBCL are moderate, suggesting each measures a different 
construct. Unexpectedly, the CAFAS and Self-Esteem measure were not significantly correlated. 
Sensitivity to change was measured using repeated measures analysis of variance and matched pair 
t-tests. There was an overall time effect for CAFAS subscales, Wilk's lambda (5,125) = 7.94, p<.001. 
Means, standard deviations, and significance tests are presented in Table 2 . 

The CAFAS Total, Role Performance, Thinking, Emotional Health, and Behavior toward Self/Others all 
showed statistically significant improvement from youths' entry into the program to one year post-entry. 
Youth in this sample also showed statistically significant improvement on the CBCL, t (126) = 4.03, 
p<.001, baseline = 71.80 (SD = 8.18), one year follow-up = 68.32 (SD = 10.40). Although statistically 
significant decreases were found on the CBCL, youth, on average, did not demonstrate clinically 
significant change on this scale. Importantly, the change in mean scores on the CAFAS suggests that 
youth demonstrated clinically meaningful change, moving from moderate toward mild impairment. 
Scores on Substance Use subscales did not show significant change. This was due most likely to the 
limited impairment in this area for the current sample. 

While mean change over time suggests promise for the CAFAS's sensitivity, mean scores do not identify 
youth who may have shown deterioration or lack of change in functioning over time. The scaling of the 
CAFAS lends itself to categorical analysis so that youth showing improvement, no change, or 
deterioration can be identified. Such analysis indicated that 62% of youth moved to a "less impaired" 
category of impairment, 1 1% showed no improvement, and 27% further deteriorat-ed over time. Percent 
improvement provides additional support for the sensitivity of the CAFAS to change over time for both 
improvement and deterioration of functioning. 

What factors account for change in CAFAS scores? It is possible that the observed change was due to 
regression to the mean, maturation, or effective provision of service. Since ARTS is not a contrasted 
group design, it is difficult to rule out maturational and regression to the mean effects. However, one can 
evaluate whether change (improvement versus deterioration) bears any statistical relationship to service 
utilization. 

Caregivers were asked whether any of 50 services, from the major service sectors as well 
as paraprofessional support, were received at any time in the first year of treatment. These 50 services 
variables were collapsed into broad-ranging categories and dummy coded as: received&shyp; yes or no. 
Point biserial bivariate correlations were computed between CAFAS at follow-up and service domain, 
partialling out baseline CAFAS score. There was a significant association between improvement in 
Substance Use and probation services (r = -.21, p<.05) and juvenile justice residential services (r = -.24, 
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p<.05). This association likely is due to restricted opportunity for these youth to use illicit substances 
while under close supervision. A significant association also was found between deterioration in overall 
functioning and receipt traditional mental health residential services (including residential treatment 
centers, group homes, psychiatric hospital; r = .28, p<.05). 

One might interpret this association several ways. It is possible that mental health residential services 
were not effective. Alternatively, the most severely impaired youth may have been offered these services 
making change more difficult to achieve. 

Findings from this study suggest that the CAFAS shows promise as a sensitive outcome measure in a 
sample of youth with serious emotional disturbance. While it is difficult to isolate the reason for change 
on the CAFAS in the present study because no comparison groups were used, these findings support 
continued evaluation of this instrument. The statistically and clinically meaningful change observed in 
this sample is especially encouraging in light of studies that 

have shown that youth with serious emotional disturbance have relatively poor outcomes (Silver et al., 
1992; Wagner, D'Amico, Marder, Newman, & Blackorby, 1991). 
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Introduction 

The aims of this summary are as follows: 

• to describe an innovative methodology for assessing children's mental health services need; and 

• to describe how this methodology can be applied in documenting the extent and determinants of 
unmet mental health services need of a sample of children receiving services in five major service 
systems. 

• To document extent and determinants of unmet need for mental health services in a sample of 
children in five major service systems as well as in the community. 

A large proportion of children with serious emotional disorder do not receive treatment. There is a dearth 
of data regarding the mental health services needs of these children (Silver, 1990). This hinders efficient, 
efficacious, and equitable mental health services provision. 

These factors were crucial in providing part of the rationale for the SED Study. A unique feature of this 
study is the study of children with serious emotional disturbance across five service settings as well as in 
a community probability sample. Selected aspects of the study methodology that are relevant for the 
assessment of mental health services need are provided below. 
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Method 

Sample 

A stratified random sample of children aged 9 to 17 years and their parent or caregiver was selected. The 
total sample size was 1,260; of these, 510 were drawn from a community probability sample and 750 
from five service systems (juvenile justice, child welfare, special education, substance use, and mental 
health). 

Measures 

The Diagnostic Interview Schedule for Children (DISC 2.3) Service Use and Risk Factors (SURF), was 
used to document aspects of the following domains: 

• demographics 

• functional impairment 
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• medical history 

• scholastic information 

• family environment 

• family management practices 

• family history of psychiatric disorder 

• pubertal status 

• verbal ability 

• instrumental competence 

• social competence 

• self-perceived competence 

• parental discord 

• service utilization and barriers (including attitudes to mental health use) 

Before there can be progress in documenting the extent and determinants of mental health services need 
of this population, it is necessary to have a methodology for describing mental health services need. 

The "Traditional" Approach 

The "traditional" approach in attempting to operationalize or quantify the need for psychiatric services in 
a service area starts with estimating the prevalence of specific disorders and associated impairment from 
epidemiological studies based on samples of children drawn from the general population. These 
prevalence rates are then extrapolated to produce estimates of the number of children in a service area 
that would be expected to meet diagnostic criteria for specific disorders. 

Disadvantages of The "Traditional" Approach Using this approach, it is not possible to relate the clinical 
and contextual characteristics of the child and family to the frequency, intensity, and types of services 
needed. It is thus not possible to 

• assess the extent to which psychiatric need is met through non-psychiatric systems (this is 
particularly relevant for the SED study since any mental health services received by the subjects 
are received in five different service systems); 

• apply the findings to various geographical areas since services may not be organized in a uniform 
manner; 

• design new types of service delivery in an efficient manner (Brewin, Wing, Mangen , Brugha, & 
MacCarthy, 1987); 

• link the assessment procedure with a framework for knowing what action should be taken when a 
particular problem is present (Brewin et al., 1987); and 

• systematically assess the degree to which individual and aggregate service needs are met. 

The Proposed Solution 

The proposed solution to the above problems in terms of assessing psychiatric need builds on the work 
of the Medical Research Council (UK; Brewin et al., 1987) and the New York State Office of Mental 
Health (Grosser, 1991). The central feature of this solution is to describe mental health services in terms 
of the types of intervention needed (Bebbington, 1990). 
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Types of intervention refers to service elements that are generalizable and comprehensible across 
different organizational units and systems of care. Examples of possible types of intervention are as 
provided in Table 1. 

Application to the SED Study 

A list of types of intervention and their definitions will be compiled. 

A list will be derived from the clinical experience of the authors, discussions with colleagues, and 
literature reviews. To estimate inter-rater reliability, 30 subjects in each system will each be rated in 
terms of their need for each type of intervention by two service providers. Each of these service provides 
will rate these 30 subjects one month later to establish test-retest reliability. The reliabilities will be 
reported as Kappa coefficients. 

Assessments of psychiatric need will be made for each child by psychiatric clinicians using data 
obtained from the SED study. 

The clinicians will make their assessments on the basis of their clinical experience, published studies 
regarding the efficacy of particular treatment modalities, and practice parameters developed by the 
American Academy of Child and Adolescent Psychiatry (Coppans, Jaffe, & Grimes, 1994). 

Prior to the assessment of the whole sample, the inter-rater reliability and test-retest reliability of the 
clinicians' use of the instrument will be established and reported as Kappa coefficients. This will be done 
by having two clinicians make judgments on 100 subjects and having one clinician make repeat 
judgments approximately one month apart on 30 subjects. 

The types of intervention received by each child will be reported by the service providers (using the 
instrument described above). 

The extent of unmet need will be calculated. 

This will be done by ascertaining which of the needs as assessed by the psychiatric clinicians are not 
being provided by the relevant service system. 

The determinants of unmet need level will be ascertained using logistic regression. 

The dependent variable will be whether a specific service was provided and the independent variables 
would comprise the individual, family, and service system characteristics. 
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Conclusions 

In this summary, we described a novel approach to the challenge of describing unmet mental health 
services need in children and we indicated how this approach would be applied in the analysis if an 
ongoing project. The data produced will be of greater use to mental health service planners than that 
available from existing methodologies. 
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Introduction 

Treatment centers face new challenges in providing services 

for children with severe emotional disturbances (SED). Given limited dollars for human services and the 
advent of managed care initiatives, they must carefully track the services each child receives, and be 
accountable for the quality and effectiveness of treatment. Too often there are gaps in essential 
information about children, interventions and outcomes. Moreover, children with SED access a variety 
of providers, both public and private. Each center has its own system for documenting care goals and 
services, but systematic communication between centers, and even within centers, is often inadequate. 

Although technology is available to help with client information needs, implementation and use of client 
information systems within human service organizations has not proven easy (see Bronson, Pelz & 
Trzcinski, 1988; Cross, Gardner & Friedman, 1993; Hallfors, Cross & Roan, 1993). 

Hastily designed systems or "canned" software packages do not necessarily provide the information 
agencies require and are often not readily adaptable to their particular information needs. Data gathering 
requirements imposed from higher administrative levels can alienate busy staff, especially if the 
information appears irrelevant to practice or redundant. Client Information Systems (CISs) are often 
implemented without adequate staff instruction in maintaining and using data. Lack of standardization 
impedes the flow of information and makes it impossible to track children across agencies for more 
powerful analyses of outcomes. 

This summary describes the first phases of development of a simple, flexible computerized Client 
Information Systems (CIS) for agencies who provide services to children with SED. The CIS will enable 
individual agencies to initiate a carefully developed core data system that they could further refine to 
meet their individual specification. It will also allow public agencies to collect standardized data about 
children across multiple systems and limit redundancy by involving the many users of data in choosing 
key elements. Providers can access information across organizational and professional boundaries to 
improve treatment. We envision a consortium of centers across the Northeast participating in large 
evaluation studies of treatment outcome and cost-effectiveness to improve service delivery. 

Our primary objective has been to create a Common Data Set structure suitable for use by any center 
providing day treatment or residential services to children. The database structure will consist of a set of 
tables, a data dictionary describing relationships among the tables, and written documentation on 
operation. We are developing this core structure in a one year project, along with a plan for field testing 
and refinement in the following year. 

Developing a CIS that can be used by many different organizations requires overcoming a number of 
obstacles. Hardware and software incompatibilities are often present, and some agencies have a strong 
commitment to a particular operating system (e.g. DOS, Macintosh, Windows). Developing software for 
one operating system is time consuming, and making that software available on a number of systems 
significantly increases the complexity of the process. Second, agencies differ considerably in how data 
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flows from one part of the organization to another, and what types of data are important for a client 
database. Considerable discussion needs to take place to discover what data users need and to establish 
consensus about what core data should be standard. The Brandeis' staffs experiences with a national 
demonstration project highlight some of the problems involved in implementing a common CIS 
software package among several organizations (see Cross, Gardner & Friedman, 1993; Hallfors, Cross, 

& Roan, 1993). 

We are suggesting a minimalist approach to the problem outlined above. Our job is to ease the transition 
from a paper-based system to an electronic CIS for participating organizations. 

By providing a comprehensive, core Common Data Set (CDS) structure, we will save time and money 
for those agencies attempting to develop their own systems, and improve their chances 
of success. A standardized CIS will allow easy access to evaluation data across agencies, yet each 
organization will have maximum flexibility in using the hardware and software of its own choice. 

In addition, researchers and the consultant are working with key personnel in the public sector, who have 
responsibility for servicing children with SED. Leaders from Massachusetts government departments 
such as public education, child welfare, children's mental health, and Medicaid have been invited to 
share their information needs and perceived trends in reporting requirements. 
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Initial Steps 

Data from Phase I of the project were drawn from five eastern Massachusetts residential treatment 
centers who serve latency age and adolescent children. All of the agencies utilize some form of 
computerized client record system representing a variety of hardware and software platforms. The 
information systems are home grown and have been through several iterations. Most were created in the 
late 1980's. The organizations are medium sized with annual budgets ranging from 4.5 to 12 million 
dollars. 

Phase I of the project focused on determining which of the hundreds of data elements used by residential 
treatment centers (RTCs) participating in the project should be candidates for inclusion in the common 
d at a set. We began by collecting and analyzing all the relevant computerized data elements that were 
tracked by the different programs. 

It proved somewhat more difficult than expected to gather the data elements from each of these busy 
agencies. Each differed in their understanding of the project and in their ability to produce the requested 
information. With perseverance, a reasonably comprehensive picture of the elements being tracked at the 
various agencies was developed. 

To this picture, data elements from the Adoption and Foster Care Analysis and Reporting Systems 
(AFCARS) published in Federal Register (Vol. 58 No. 244) were added. AFCARS identifies data 
elements that states will be required to report to the federal government in an initiative to produce 
national information on foster care and adoption. These elements were added to determine the extent 
which the final common data set would be AFCARS compliant. However, the addition was not as useful 
as originally hoped, given that the AFCARS elements represent primarily derived data or the results of 
initial data analysis. In addition, a core dataset identified by the American Association for Partial 
Hospitalization (AAPH) Outcomes Measurement Protocol has been entered into the application. The 
core data set represents a collection of instruments including outcome measures. At the time of this 
report, these data elements have not been analyzed. 

The elements provided by treatment centers were entered into a comparison application. Elements were 
given a common element name which captured its intended function (see Table 1). Elements were later 
reviewed and grouped into three categories: core data set, administrative, and other. 

To be included in die core data set, elements needed to address the following key questions: What kind 
of children are served? What kind of services do they receive? When do they enter and leave the 
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program? Who pays for their care? How do they do after they leave? 

As shown in Table 1 , only a few elements currently collected can answer these questions. Most of the 
elements respond to administrative or management requirements. We plain to share these elements with 
RTCs for their consideration, but would not recommend them for a core data set. For the CDS, we 
pulled what we could from the list in Table 1, and then considered additional variables that could answer 
our questions. 

Table 2 displays all of the elements being considered for the CDS. Two items deserve special attention. 
For example, we are developing a unique identifier that, while protecting children's confidentiality in 
analysis and reporting, will allow them to be tracked throughout their stay, as well tracking returns to 
this RTC or any other. Ideally, we will develop the ability to track children throughout the health and 
welfare service system. The identifier may include markers that would allow family members to be 
recognized in some way. Services to siblings or other family members then could be analyzed without 
divulging confidentiality. Social security numbers are currently used to validate the right record for the 
right child. Social security numbers mask a child's identity to some degree, but require additional 
security to protect confidentiality. 

We are also paying special attention to the outcomes variables. The ones listed in this paper have caused 
us to ask additional questions. For example, how will the outcome variables be used? Will RTCs be 
compared for their outcomes? Will attention to outcomes make RTCs avoid the most problematic 
children? Measuring outcomes has become quite fashionable, but each outcome indicator must be 
considered carefully for the conclusions it implies. 
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The Next Steps 

During the next phase of the project, we will develop an advisory committee to review all data elements 
and make recommendations for further changes in the core data elements. The advisory committee will 
consist of representatives from the RTCs, state agencies, consumers and researchers. All elements will 
be subject to discussion and debate over inclusion in the CDS. Those variables that finally are included 
will then be defined and operationalized for programming in a relational data base. The data base will be 
piloted in six RTCs and further refined by the advisory committee as needed. We envision this to take 
place over a two-year period, at which point we will make the data set available for a much wider circle 
of RTCs. 

Return to Top 



References 

Bronson, D., Pelz, D., & Trzcinski, E. (1988). Computerizing your agency's information system. 
Newbury Park, CA: Sage Publications. 

Cross, T. P., Gardner, J. K., & Friedman, C. (1993). Implementation of a clinical management 
information system for children's mental health care: Lessons from a national demonstration. Research 
and Evaluation in Group Care, 3, 10-13. 

Hallfors, D., Cross, T. P., & Roan, C. (1993). Case Data (MIS). Section Three in the Evaluation of the 
Mental Health Services Program for Youth: Interim Report to the Robert Wood Johnson Foundation. 
Boston, MA: Brandeis University, Heller School, Family and Children's Policy Center. 

Return to Top 



Denise Hallfors, Ph.D. 

Senior Research Associate 
Family and Children's Policy Center 



me 



41 



12/11/98 1:41 PM 



8th.365 

t ft 


http://lumpy.fmhi.usf.edu/CFSroot/rtc/proceeding8th/8th.365.html 


Heller School 
Brandeis University 
Waltham, MA 02254-9110 
617/736-3946 Fax: 617/736-3851 

Michael Leach, M.P.A. 

Business Manager 
Germaine Lawrence School, Inc. 
18 Clairmont Avenue 
Arlington, MA 02174 
617/648-6200 Fax: 617/648-9106 

Floyd Alwon, Ed.D. 

Director 

Albert E. Treischman Center 
1968 Central Avenue 
Needham, MA 02192 
617/449-0626 Fax: 617/449-9074 

Return to Top 





fr|c 


4 2 

12/11/98 1:41 PM 



12/11/98 1:41 PM 



,365tl http://lumpy.tinhi.usf.edu/CFSroot/rtc/proceeding8th/8th.365tl 









CO 




12/11/98 1:16 PM 



B 

d 

yo 

m 

6 

00 



00 

oo 

.9 

■o 

<u 

D 

o 

o 



0 

1 

o 

o 

t- 

00 

Uh 

u 



•o 

<D 



s 

CL. 

00 



00 

p's 



CD 



CN 






C/1 

3 







I 




LO 



Cl 

ir> 




O 



8th.37f.html 



http://lumpy.fmhi.usf.edu/CFSroot/rtc/proceeding8th/8th.37 1 .html 



I Home I I search 1 1 index"! 

8th Annual Research & Training Center Conference Proceedings, Dept of Child and Family Studies, 
Florida Mental Health Institute, University of South Florida, 1 996 

Enhancing the Methodology of Social Validation: 

A Survey of Child and Family Service Issues 

Authors 



Introduction Method Results References 
Back to Table of Contents 



Introduction/Purpose 

Social validity relates to the idea that behavioral interventions and outcomes must be validated by 
individuals and communities being served. Social validation research, and the related, but more public 
policy oriented, Community Concerns Report Method represent a simple, direct methodology for asking 
customers what their needs are and how satisfied they are with current services (Schriner & Fawcett, 
1988). This approach evolves from a body of behavioral research on social validation that first made the 
case for subjective measurement of behavioral interventions (Kazdin, 1977; Wolf, 1978). 

As the scope of social validation research broadens to include public policy issues, questions remain 
regarding reliability and validity (Fawcett, 1991; Storey, 1992; Storey & Homer, 1991). The Southern 
Consortium for Children (SCC) Survey of Child and Family Service Issues builds upon the work of the 
Pennsylvania Project Social Validation Survey (VanDenBerg, Beck, & Howarth, 1992), by adapting a 
previously used survey instrument for field testing in Appalachian Ohio. 

The purpose of the study was to apply appropriate psychometric tests to data already collected by the 
Pennsylvania Survey, to alter items where indicated, and to pilot the resulting instrument in four Ohio 
counties in an attempt to evaluate reliability and validity. The resulting instrument is available for use 
elsewhere, encouraging more uniform methodology in further social validation research. The study was 
carried out in two distinct phases. 
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Methodology and Findings, Phase One 

Overview of Pennsylvania Project Data 

The Pennsylvania Survey was designed to obtain some measure of the opinions of stakeholders in 
Pennsylvania's children's services system. Specifically, this survey asked stakeholders to rank the level 
of importance of selected children's issues and their satisfaction with how these issues were addressed. 
The goal was to identify issues of both high importance and low satisfaction, and to target those issues 
for the development of program outcome measures. 

Seven hundred Pennsylvania stakeholders were asked a series of questions regarding the importance and 
satisfaction levels associated with various children's issues. Children's issues were categorized into the 
following life domain areas: educational, vocational, safety, living arrangements, family life, emotional 
and psychological, medical and psychiatric, social and recreational, cultural, spiritual, and legal issues. 
Respondents chose 

from among values on a Likert type scale for both importance and satisfaction levels on 49 questions, 
effectively yielding 98 separate survey items. For all questions, the choices were 1, 2, 3, 4, and 5, with 1 
being the least important or least satisfied and 5 being the most important or most satisfied. 

Ten specific groups of individuals were identified as stakeholders of children's services. The ten 
identified groups were: children age 12 or older, parents of children younger than 12, parents of children 
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age 12 or older, community agency workers, board members of agencies, school board members, 
juvenile court judges, local legislators, informal community leaders, and poverty-level representatives. 

Phase One Procedures 

In phase one of the study, data collected in the Pennsylvania Survey were secured so that an item 
analysis could be performed in an attempt to obtain a meaningful measure of instrument reliability. 
Specifically, internal consistency was measured using Cronbach's (1951) coefficient alpha, which is a 
commonly accepted formula for assessing the reliability of a measurement scale with multi-point items. 
Survey items were split into life domain areas and further along the dimensions of importance and 
satisfaction, yielding 22 separate sets of data. Within each data set, a coefficient alpha value was 
determined for each set when one item was deleted, one item at a time. This allowed for the 
identification of items that significantly eroded internal consistency. 

Further statistical analysis was carried out to assess the significance of reported variability between child 
clients and all other respondents, and between groups closest to services and those groups one step 
removed from services. An independent groups t-test (Jaccard & Becker, 1990) was performed 
comparing the mean ratings for each of the comparison groups described above. 

Phase One Findings 

Given the fact that the survey instrument is in an early stage of development and there were a small 
number of items in each life domain data set, alpha values of .6 across the dimension of importance and 
.7 across the dimension of satisfaction were considered to be adequate indicators of instrument 
reliability. This was found to be the case in 1 5 of the 22 data sets, which was considered to be evidence 
in support of reliability. The instrument was found to be reasonably reliable, in general. Life domain 
areas that did not meet the standard across both the dimensions of satisfaction and importance were 
selected for alteration, while those that exceeded the standard were considered to be reliable and were 
included in the instrument for field testing in Ohio without changes. The Pennsylvania Project staff also 
suggested two minor changes based on previous experience. In all, at least one item within 8 of the 11 
life domain areas was selected for alteration in phase two of the study. 

It was intuitively felt that an instrument of reasonable reliability would be able to discriminate between 
child and non-child respondent groups, and between those closest to services and respondent groups one 
step removed from services. A positive finding in either or both instances was to be considered evidence 
in support of construct validity. 

When independent groups t-tests were performed on both comparison groups, results were significant. 
Statistically significant differences were discovered in 29 of 44 data 

sets, overall. The direction of significant differentiation was very consistent between respondent groups 
across life domain areas. Evidence in support of construct validity was found to be present in phase one 
of the study. 
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Methodology and Findings, Phase Two 

Phase Two Procedures 

Stakeholders of child and family services were selected for participation in phase two of the study in 
Athens, Jackson, Washington and Lawrence Counties in rural, southeastern Ohio (N = 192). Stakeholder 
groups were the same as the Pennsylvania Project Survey with the exception of the poverty level 
representatives, who were excluded at the suggestion of Pennsylvania Project research staff. 

It was decided that when at least 60 valid responses were received, the statistical analyses described in 
phase one would be repeated and a follow-up interview would be conducted with 10 individuals. 
Interview candidates were nominated by mental health agency workers and represented all stakeholder 
groups, with the exception of judges. Independent groups t-tests were performed only where significant 
differences between groups were discovered in phase one of the study. 

Phase Two Findings 
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The minimum standards for values for Cronbach's alpha established in phase one of the study were 
exceeded in all but 2 of 22 life domain data sets in phase two. Values for alpha were increased in 8 of 1 1 
instances across the dimension of satisfaction, and in all instances across the dimension of importance. 

Independent groups t-tests were performed using pooled variance estimates on nine data sets where 
child-client ranking of satisfaction levels significantly differed from other groups in phase one. There 
were significant differences in four of nine data sets. 

Independent groups t-tests were also performed on the eight data sets where significant differences were 
discovered in phase one between child clients and all others in their ranking of importance. There were 
significant differences in two of eight data sets. 

Similarly, independent groups t-tests were performed on respondent groups closest to services, and 
groups one step removed from services in ranking of importance and satisfaction where significant 
differences were present in phase one data. There were no significant differences found in any of the 
seven areas when ranked for satisfaction. When ranked for importance, a difference was found in only 
one of five areas examined. 

When semi-structured follow-up interviews were conducted among 10 survey respondents, a high degree 
of satisfaction was reported with the content, clarity and significance of the questionnaire. When 
specifically asked about the importance of the life domain areas chosen for study, they were universally 
endorsed. The follow-up interviews yielded no significant negative comments. 

Return to Top 



Summary & Conclusions 

An increase in values for Cronbach's alpha in 20 of 22 life domain data sets represents compelling 
evidence in support of enhanced instrument reliability for the altered questionnaire that was field tested 
in Ohio. However, the performance of independent groups t-tests in phase two, failed to convincingly 
replicate the phase one study. Little new evidence in support of construct validity was yielded in phase 
two. The follow-up interview suggests a high degree of overall respondent satisfaction with the 
instrument, and the one specific question regarding the importance of included life domain areas 
provides an indicator of content validity. 

It is expected that enhanced methodology in social validation research will increase its viability and help 
to more clearly place this approach in the larger context of health and human service policy 
development. Specifically, it is anticipated that the application of psychometric measures to survey data 
will help to increase consumer participation in the design of child and family programs in Ohio and 
beyond. 

The SCC is dedicated to an outcome manage-ment approach that includes consumer choice and 
contributes to the overall responsiveness of services. Working closely with families, service providers 
and community groups, the Survey is being used as part of a backdrop for the development of relevant 
outcome measures across systems and life domain areas. 
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Introduction and Purpose 

The purpose of this project was to design effective and efficient outcome measures for a youth partial 
hospitalization (YPH) program. The design was required to utilize information obtained through the 
existing intake structure, as well as to provide immediate information that could assist in treatment 
decisions, treatment planning and the design of interventions. The resulting design must also obtain 
information across several sources and serve as a means for monitoring clients’ progress. Additionally, 
the instrument to be selected required appropriate standardization and must be acceptable for use in the 
YPH setting. Finally, it needed to be cost effective and easily incorporated into the established role of 
the therapist. Once the design was developed, a pilot was initiated. This summary presents the design of 
the outcome measurement process and results of the first six months of data collection. 
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Method and Procedures 

The design described in this summary was developed and implemented in a partial hospitalization 
program setting which serves youth who experience significant emotional and behavioral difficulties. 
The YPH setting is part of a larger community mental health system of care in Dayton, OH. Individuals 
who are referred to this program demonstrate severe symptoms of emotional or behavioral disorders 
which interfere with successful functioning within the structure of the school, home and/or the 
community, however they do not require hospitalization. Treatment goals include preparing the youth 
for return to community settings with behaviors that are manageable and appropriate to the situation. 
Therapy includes work in groups, in individual sessions, and family sessions. Academic instruction is 
provided daily through a tutoring model. 

The Achenbach Child Behavior Checklist (CBCL) was selected to obtain behavioral data due to its 
multimodal and multi-informant format. The instrument met design criteria, because it can be 
administered to parents, teachers and youth, yields a cross-informant profile for response comparison, 
and has been standardized (Achenbach, 1991). The results provide a profile that delineates internalized 
behavior (e.g., with-drawn, somatic complaints and anxietydepression) from externalized behavior (e.g., 
delinquency and aggression). Social problems, thought problems and attention problems can also be 
identified. The scales are easy to complete, and results can be compiled quickly with the aid of computer 
scoring. It was felt that it would be feasible to incorporate information provided by the CBCL into 
treatment planning and intervention design. 

Client information from the initial intake 

and existing client records contributed to the comprehensiveness of the outcome measures. The 
following variables were considered: gender, ethnicity, age at admission, initial primary diagnosis, 
initial secondary diagnosis, most frequent medications used, length of stay, global assessment of 
functioning (GAF), and use of additional services. 
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Clinically, it was expected that three specific areas would be included in the development of acceptable 
outcome measures: level of functioning, severity of symptoms, and client satisfaction. Table 1 describes 
the resulting evaluation design. 

Two data collection points were established. The first occurred at admission and the second during a six 
month review. The data were analyzed using a paired samples T-test to determine whether significant 
change occurred during the first six month of treatment. Results from the CBCL were compared to 
determine specific areas of significant change. 
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Results 

The results of the paired samples T-Test for the initial and six month data provided insight into the 
effectiveness of the YPH intervention. The parent or primary caretaker reported a significant reduction 
in the youths' anxious/depressed behavior, aggressive behavior and externalizing behavior (see Table 2). 
The teachers working with the child in the treatment milieu reported a decrease in externalizing (acting 
out) behavior. The most significant finding related to the youth's perception of their own behavior. 

Youth reported that after six months they experienced less withdrawal, a decrease in social problems and 
a decrease in attending problems. The youth self report total scale indicated significant changes across 
these measures. 

The initial placement profile that resulted from the collection of demographic data is shown in Table 3. 
The initial profile of the client was considered in concert with the CBCL measures. The CBCL's 
Internalizing and Externalizing scales helped to identify specific behaviors related to the diagnosis, 
allowing the treatment plan and intervention selection to be directly related to the symptomatology. 
Initially, it was anticipated that Attention Deficit Hyperactive Disorder (ADHD) would be the most 
common primary diagnosis. The results of the pilot study, however, identified Dysthymia as the most 
frequent primary diagnosis, and Oppositional Defiant Disorder as the most frequent secondary diagnosis. 
It appeared that therapeutic intervention for depression might be as important as redirecting the 
externalizing behavior. Considering the most frequent primary and secondary diagnoses, the most 
frequent medications used were as expected&shyp;Prozac (anti-depressant) and Ritalin, often prescribed 
to address externalizing behavior. These results suggested that a combination of interventions was 
necessary to promote positive outcomes, including psychotherapy, behavior change and medsomatic 
services. Results also supported the utility of a transdisciplinary approach to service provision. 

The 180 day length of stay (LOS) was identified as the baseline measure for program planning. To 
impact LOS, increased services to support the successful transition of clients to the least restrictive 
appropriate environment must be developed. Case management and foster care services are utilized in 
conjunction with YPH program within the system of care model, and can help support these transitions. 
Collaboration with the schools will also be a critical link to accomplish this goal. 

Additionally, the results indicated that the lower the score on the GAF upon intake, the longer the length 
of stay. This information will help predict the level of financial support required from funding sources 
for individual clients. In future studies, it is anticipated that examination of these factors will yield an 
anticipated LOS upon intake. 
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Discussion 

It is likely that comprehensive services for youth who have severe emotionally disturbances support the 
ultimate goal of preventing future inpatient hospitalization and residential care. 

The YPH program will use results of the outcome measures to focus their efforts in making the 
continuum of care accessible to children and adolescents in need of mental health services. Program 
standards will be reviewed to ensure that age appropriate and/or developmentally appropriate types of 
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services are made available. Skill groups, for example, could meet the needs of a specific age group 
(e.g., sex education, anger management, or independent living skills). 

The initial results of this study have had a great impact on refining the direction of the YPH program. 
The information from this study will be used in both treatment planning and program development, and 
should gamer agency support of the outcome measure process. Ultimately, information obtained from 
outcome measures will be critical in the justification of managed care dollars and authorization of 
services for children, adolescents and their families. 
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Introduction/ Purpose 

As systems for care for youth with serious emotional disturbances are designed and implemented, 
deficiencies of the existing methods of service planning have been identified. Some methods use 
language that is unfriendly to families&shyp;medically oriented categorical diagnoses, for example, 
create barriers for families, and may not guide appropriate professional intervention. Some are 
operationally explicit, but do not accommodate or embrace a wide range of clinical approaches. 

Some methods of service planning do not take into account the purposes of different systems (e.g., child 
welfare vs. special education). Because of these shortcomings, systems of care, despite efforts to achieve 
structural integration, often find it is difficult to achieve functional integration&shyp;that is, operational 
approaches to service planning that can be used throughout and across a system of care. 

This summary presents evaluations of a method of service planning that responds to these problems. 

This method was designed to be operationally explicit, family-friendly, conducive to the integration of 
clinical experience and multi-agency competency into service planning, and use of a language of service 
planning that will "work" across systems. 

Focal, factor-based service planning (FFSP) was developed in a hospital setting and has been used in a 
community-based agency (Harper, 1989; Harper & Cotton, 1991). FFSP assumes that service planning 
can be explicit and operational, comprehensive and coherent. It's design is based on the assumption that 
it is possible and preferable for parents and youth to participate in the development of the service plan. 

In addition to family participation in planning, FFSP values the use of least restrictive alternatives, and 
parsimony&shyp;the definition of modest goals at each stage&shyp;doing what needs to be done, not all 
that could be done. 

FFSP begins with selection, from among many potential problems, of a single Focal Problem, which is 
explicit, actual, and defined in terms that all parties can agree on. A corresponding Goal is then defined. 
Contributing Factors, those elements in the life of child, family, community or systems of care felt to 
contribute to the Focal Problem, are identified, and Intervenable Factors chosen. Corresponding 
Interventions, Objectives and Target Dates can then be defined. The Focal Problem and Goal and the 
Selected Factors must be stated in language that can be shared with the youth and with parents. The 
overall plan is guided by an explicit Discharge Picture. Criteria for Focal Problem, Goal, and Factors are 
given in Table 1: questions to use to check on the plan are given in Table 2. 

The Focal Service Plan does not include everything known about the child; it complements and does not 
take the place of clinical diagnoses; and is not a complete problem list. 

This summary presents results of the evaluation of FFSP in hospital and community settings and 
suggestions for implementing its use in systems of care. 
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Method/ Procedures 

For this evaluation research, two sites were chosen. One, an inpatient psychiatric service in a teaching 
children's hospital, had used FFSP (there, called Focal Inpatient Treatment Planning, FITP) for several 
years (Flarper, 1989; Harper & Cotton, 1991). The other, a community-based family service agency, had 
begun using FFSP in the previous year. The staff and trainees in these two sites represented a wide range 
of experience and came from multiple disciplines. 

The evaluation questions were, (1) What perceived advantages does FFSP have for clinicians? (2) What 
is the impact on clients? Is FFSP relevant to long-term outcome? (3) How do experienced and trainee 
clinicians learn to use FFSP? and (4) How is the method used in multidisciplinary meetings? Data were 
gathered through observations of team meetings and semi-structured interviews with clinicians by one of 
the authors (J. Barnes). 
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Results and Implications 

Observations in team meetings and comments in interviews indicated that FFSP was indeed being used, 
both in home-based and inpatient services. It was experienced as useful in both settings, for example, in 
identifying the unique aspects of each case, and decreasing the risk of "losing the forest for the trees" in 
complicated cases. Respondents reported that they appreciated having a language that could be shared 
among family members and providers from different disciplines. 

FFSP was reported to have empowered clients by stating the work to be done 

in terms that youths and parents could understand and by emphasizing strengths and reachable goals. It 
stated mental health problems in non-pejorative language that could be shared with other family 
members. It was felt to facilitate work between clients and clinicians, and to clarify targets for other 
professionals. 

Moving beyond these preliminary, subjective assessments, issues for further research included 
determining optimum training strategies; finding the best methods for generalizing use of FFSP between 
settings; assessing clinician satisfaction and advantages for clients in a more systematic way; and 
evaluating the contribution of FFSP to longer-term implementation of goals. Semi-structured interviews 
with clinicians differing in experience, discipline, and setting, interviews with children and parents, and 
observations in teams will help to address these issues. 

In the meantime, the use of FFSP will respond to external incentives. There is community pressure for 
participation in planning, while accrediting and reimbursing agencies increasingly require evidence of 
multi-disciplinary treatment planning with explicit goals and objectives. The use of the FFSP 
methodology should also increase in response to internal incentives&shyp;such as care providers' 
frustration at getting "bogged down" in data-rich cases where a goal is difficult to define, and their desire 
to pool input from professionals and community members alike. 

As to implementation, our observations suggest several necessary conditions. On the institutional level, 
these include a mandate for explicit, participatory service planning; and the provision of needed 
resources, in terms of time, training, hardware, and the development of forms suited to this kind of 
service planning. Among personnel, implementing FFSP requires openness to learning new ways to 
organize clinical data, perhaps divergent from traditional practice; the capacity at each phase of the work 
to step back and reflect on the planning process itself; and a commitment to consensus development and 
participatory service planning. 

Implementation requires that staff development be thought of as a process, not as a one-time training 
session; that it emphasize learner-directed, experiential learning; and de-emphasizes paperwork, or 
"getting the right form" as the solution to the challenges of service planning. Training must address 
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critical knowledge, skills, and attitudes. Staff must learn that categorical diagnoses do not determine the 
content of treatment; they must learn how to use criteria for defining the problem being treated, the 
contributing, and the intervenable factors; and they must learn that obfuscation in service planning is not 
only a problem in documentation, but in clinical practice as well. They must become skilled in 
monitoring the planning process, listening at each phase for the degree of consensus, and in defining 
factors, especially intervenable factors, out of clinical data. They must adopt the values of critical 
listening, parsimony in planning, and participatory planning. 
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Introduction and Definition 

This summary explores the use of medical anthropology models to explain the relationships between 
parents of children labeled seriously emotionally disturbed and service providers. Today the values and 
practices of community service providers are moving away from institutionalization and toward 
community care and "individualized service models" (Bums & Friedman, 1990:95). This shift includes 
the recognition that the family of a child with a serious emotional disturbance requires as much support 
as does die child. Of equal importance, parents are becoming recognized and valued as the experts in 
their children's problems and needs, and are beginning to participate in service planning for their 
children. 

In the new individualized service model, "the concept of a partnership between parents and professionals 
expands the more traditional roles of 'patient' and 'client' the parents tend to assume" (Friesen & 
Koroloff, 1990). As this service model becomes more accepted by all services and providers in the 
children's mental health service systems, a better understanding must be gained about what is happening 
in the relationships between the parents and providers. 

Medical anthropology has a number of established theoretical models that address the relationship 
between patients and healers, which are used here to examine this relationship in the children's mental 
health service systems. Medical anthropology models are sociocultural models that recognize that the 
system's context is necessary in studying the strategies a family uses to seek and obtain care and support 
for a family member with medical or psychological health problems. 
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Method 

Medical anthropology models were used to analyze families' strategies for seeking and interacting with 
the available services, such as the special education schools explored in the case study. Data analyzed 
were from a qualitative multiple-case study of two families. In depth interviews, conducted in 1992, with 
young men and mothers explored their interactions with the schools and other service providers. The 
families were participants in the National Adolescent and Child Treatment Study (NACTS), a research 
project conducted by the Research and Training Center for Children's Mental Health, Department of 
Child and Family Studies, Florida Mental Health Institute (FMHI), University of South Florida (USF). 

Anthropology of Biomedicine 

The Anthropology of Biomedicine model is primarily cognitive. This approach and model "focus on 
what sickness and health mean to individuals and how, as a consequence of these meanings people act in 
specific ways regarding treatment" (Lazarus, 1988). Kleinman (1980), had developed an explanation for 
the anthropology of biomedicine model. He uses the concept of "explanatory models", which are notions 
held by the patients, families and practitioners about specific episodes of illness (Kleinman, 1988). 
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Explanatory models may include cognitive, affective, moral and social concerns (e.g., specific health 
beliefs, fears and concerns about health, stigma of SED label, etc.) 

For the child with a serious emotional disturbance, the lack of mutual understanding on the part of 
his/her family and the school personnel can cause miscommunication resulting in long-term 
consequences for treatment decisions and outcomes. The Anthropology of Biomedicine model is well 
suited to explain miscommunications and family noncompliance with the school personnel's treatment 
recommendations. 

EXAMPLE - Case Study A.L Mother, Dana Mae Dodd; Son, Chuck Dodd; and Special Education 
Teacher, Ms. M. 

Dana Mae Dodd and Ms. M. struggled to understand and work with each other. They each had their own 
perspective and explanatory model. 

Dana Mae expected to have a lot of say and influence in how Chuck was treated at the day treatment 
facility. She kept in close contact with the teachers and expected to hear about problems on a daily basis, 
if needed. Dana Mae was assertive, she said "... I got what I wanted because I demanded it. I didn't ask, I 
demanded it when it came to my young-un...over the years the teachers and I have learned to work 
together." 

Ms. M. explained how irritating Dana Mae's behavior could be when she called constantly and sent notes 
to school that were not written clearly. Ms. M. acknowledged that Dana Mae had her good points, 
saying, "She seems genuinely concerned. He [Chuck] does get his medicine when things get serious 
enough. ... She is better than most of the parents. I see some of the same behaviors in Chuck as I do in 
Dana Mae." 

Dana Mae's belief that she should have daily input into Chuck's treatment often clashed with Ms. M.'s 
beliefs on how best to teach Chuck. This led to tension between the two women and problems in 
providing Chuck the best care. 

Critical Medical Anthropology 

The Critical Medical Anthropology model incorporates "political and economic determinants of society 
into studies of health and medicine" (Lazarus, 1988:45). The personal experience is seen as taking place 
in the social context of the society as a whole . Health is defined in this model "as access to and control 
over the basic material (e.g., transportation, medication) and non-material resources (e.g., education, 
health information, therapy) that sustain and promote life at a high level of satisfaction" (Baer, Singer, & 
Johnsen 1986). 

The critical medical anthropology model places the family-service provider interactions in the context of 
the children mental health service systems. The context influences the available services, beliefs about 
mental illness in children, and the "asymmetrical social relations based on social class and authority", 
which all become critical parts of the analysis and model (Lazarus, 1988). 

EXAMPLE - Case Study B: Mother, Beth Brennan; and Son, Calvin Brennan 
Beth wanted to know what was wrong with Calvin. At one time she had wanted him to have a 
neurological examination. She said "I wanted the examination of the brain, the head, to see if, where the 
problem was coming from. If it was something mental, psychological, something I did, something I did 
when I was pregnant ... or anything. I wanted more help. Medical help or advice which we didn't get." 

She explained why they did not get more help: "Money I guess. ... Nobody ever recommended it. It 
wasn't paid for through the school and we didn't have the money to do it ourselves. His insurance didn't 
cover stuff like that." The asymmetrical social relationship between the school personnel and Beth meant 
that Beth felt she did not have the right or knowledge to question the actions of the teachers and 
administrators. Calvin never did get a neurological exam because of economic, communication, and 
service system barriers. 
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Summary and Implications 

In the children's mental health service system there is growing recognition that there must be a true 
partnership between the parents and service providers. There is also a conceptual shift towards families 
of the mentally ill in terms of "a model of stress, coping, and adaptation [that] views familial behaviors 
as coping strategies," rather than as interfering with prescribed treatments, or as toxic agents (Lefley, 
1989). 

Medical anthropology models are well suited to examine what happens within the relationships 
developed between the families of children with mental health problems and the services providers, e.g. 
special education schools. Of particular interest was using the models to examine the family and school 
personnel interactions and their ability to cooperate in the care of the child. The models are also useful in 
exploring the reasons why the family and child did or did not comply with the treatments and 
instructions provided by the schools. 
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